A versatile stochastic model of a function of unknown and time varying form

AbstractProperties of a random walk model of an unknown function are studied. The model is suitable for use in the following (among others) problem. Given a system with a performance function of unknown, time varying, and possibly multipeak form (with respect to a single system parameter), and given that the only information available are noise perturbed samples of the function at selected parameter settings, then determine the successive parameter settings such that the sum of the values of the observations is maximum. An attempt to avoid the optimal search problem through the use of several intuitively reasonable heuristics is presented

Stochastic Approximation Methods for Systems Over an Infinite Horizon

The paper develops efficient and general stochastic approximation (SA) methods for improving the operation of parametrized systems of either the continuous- or discrete-event dynamical systems types and which are of interest over a long time period. For example, one might wish to optimize or improve the stationary (or average cost per unit time) performance by adjusting the systems parameters. The number of applications and the associated literature are increasing at a rapid rate. This is partly due to the increasing activity in computing pathwise derivatives and adapting them to the average-cost problem. Although the original motivation and the examples come from an interest in the infinite-horizon problem, the techniques and results are of general applicability in SA. We present an updating and review of powerful ordinary differential equation-type methods, in a fairly general context, and based on weak convergence ideas. The results and proof techniques are applicable to a wide variety of applications. Exploiting the full potential of these ideas can greatly simplify and extend much current work. Their breadth as well as the relative ease of using the basic ideas are illustrated in detail via typical examples drawn from discrete-event dynamical systems, piecewise deterministic dynamical systems, and a stochastic differential equations model. In these particular illustrations, we use either infinitesimal perturbation analysis-type estimators, mean square derivative-type estimators, or finite-difference type estimators. Markov and non-Markov models are discussed. The algorithms for distributed/asynchronous updating as well as the fully synchronous schemes are developed

Heavy Traffic Analysis of AIMD Models

The goal of this paper is to study heavy traffic asymptotics of many Additive Increase Multiplicative Decrease (AIMD) connections sharing a common router in the presence of other uncontrolled traffic, called "mice". The system is scaled by speed and average number of sources. With appropriate scalings of the packet rate and buffer content, an approximating delayed diffusion model is derived. By heavy traffic we mean that there is relatively little spare capacity in the operating regime. In contrast to previous scaled models, the randomness due to the mice or number of connections is not averaged, and plays its natural and dominant role. The asymptotic heavy traffic model allows us to analyze buffer management policies of early discarding as a function of the queue size and/or of the total input rate and to choose its parameters by posing an appropriate limiting optimal control problem. This model is intuitively reasonable, captures the essential features of the physical problem, and can guide us to good operating policies. After studying the asymptotics of a large number of persistent AIMD connections we also handle the asymptotics of finite AIMD connections whose number varies as connections arrive and leave. The data illustrate the advantages of the approach

Approximation and Limit Results for Nonlinear Filters Over an Infinite Time Interval: Part II, Random Sampling Algorithms

The paper is concerned with approximations to nonlinear filtering problems that are of interest over a very long time interval. Since the optimal filter can rarely be constructed, one needs to compute with numerically feasible approximations. The signal model can be a jump-diffusion, reflected or not. The observations can be taken either in discrete or continuous time. The cost of interest is the pathwise error per unit time over a long time interval. In a previous paper of the authors [2], it was shown, under quite reasonable conditions on the approximating filter and on the signal and noise processes that (as time, bandwidth, process and filter approximation, etc.) go to their limit in any way at all, the limit of the pathwise average costs per unit time is just what one would get if the approximating processes were replaced by their ideal values and the optimal filter were used. When suitable approximating filters cannot be readily constructed due to excessive computational requirem..

Some Estimates for Finite Difference Approximations

Some estimates for the approximation of optimal stochastic control problems by discrete time problems are obtained. In particular an estimate for the solutions of the continuous time versus the discrete time Hamilton-Jacobi-Bellman equations is given. The technique used is more analytic than probabilistic

A fully-discrete scheme for systems of nonlinear Fokker-Planck-Kolmogorov equations

We consider a system of Fokker-Planck-Kolmogorov (FPK) equations, where the dependence of the coefficients is nonlinear and nonlocal in time with respect to the unknowns. We extend the numerical scheme proposed and studied recently by the authors for a single FPK equation of this type. We analyse the convergence of the scheme and we study its applicability in two examples. The first one concerns a population model involving two interacting species and the second one concerns two populations Mean Field Games